Summary

Adds TsFile 2.x table-model as a real-dataset format alongside CSV, selected by a new
REAL_DATASET_FORMAT=CSV|TSFILE config (default CSV). Works on both sides:

• Generate (generateDataMode): exports the synthetic dataset as TsFiles (one .tsfile per
  (client, table) under FILE_PATH/<table>/data_<clientId>.tsfile) plus an info.txt snapshot.
• Verify (verificationWriteMode / verificationQueryMode): replays a TsFile dataset, streaming
  batches grouped by device.

Semantic mapping: a benchmark device = a device_id TAG value within a table; FIELD columns =
sensors; TIME column = record timestamp. core gains an org.apache.tsfile:tsfile:2.2.1 dependency
(aligned with what iotdb-2.0 already resolves). CSV / Copy paths are unchanged.

Key components (all in core):

• conf/RealDatasetFormat + REAL_DATASET_FORMAT wired through Config/ConfigDescriptor/config.properties
• source/TsFileTableModelMapping — SensorType↔TSDataType + column classification
• source/TsFileSchemaReader, source/TsFileDataReader (read); extern/TsFileSchemaWriter, extern/TsFileDataWriter (generate)
• Format dispatch in DataReader.getInstance, RealMetaDataSchema (file-level distribution), SchemaWriter.getBasicWriter, DataWriter.getDataWriter
• SchemaReader.checkDataSet lifted to the base (shared by CSV + TsFile) and hardened against a shorter info.txt
• GenerateDataWriteClient closes the writer on finish (TsFile footer)

Try it locally (generate → replay)

Two phases: generate writes a TsFile dataset to disk (no DB required); verify replays it into a
running IoTDB 2.0 (table model) with write/query verification.

Build

mvn clean package -pl iotdb-2.0 -am -Dmaven.test.skip=true
cd iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0

Phase 1 — generate the TsFile dataset (no DB needed). In conf/config.properties:

BENCHMARK_WORK_MODE=generateDataMode
REAL_DATASET_FORMAT=TSFILE
FILE_PATH=data/tsfile-demo
# dataset shape — these go into info.txt and MUST match in phase 2
DEVICE_NUMBER=5
SENSOR_NUMBER=10
LOOP=100
BATCH_SIZE_PER_WRITE=100

Run ./benchmark.sh. Output (ls -R data/tsfile-demo):

data/tsfile-demo/
  info.txt                       # config snapshot, validated on replay
  <table>/data_<clientId>.tsfile # one TsFile per (client, table); devices keyed by device_id TAG

Phase 2 — replay into IoTDB 2.0 (table model) + verify. Start an IoTDB 2.0 instance, then in the
same config.properties change only:

BENCHMARK_WORK_MODE=verificationWriteMode
DB_SWITCH=IoTDB-200-SESSION_BY_TABLET
IoTDB_DIALECT_MODE=table
HOST=127.0.0.1
PORT=6667
USERNAME=root
PASSWORD=root

Run ./benchmark.sh (write verification); then set BENCHMARK_WORK_MODE=verificationQueryMode and run
again (query verification).

Note: the dataset-shape params (REAL_DATASET_FORMAT, DEVICE_NUMBER, SENSOR_NUMBER, LOOP,
BATCH_SIZE_PER_WRITE, START_TIME, …) are recorded in info.txt and compared by checkDataSet()
on replay — keep them identical between the two phases (only BENCHMARK_WORK_MODE and the DB
connection settings change). DB connection settings are not part of info.txt.

Robustness — abnormal-config / malformed-dataset handling

Hardened against NPEs surfaced while running the flow end to end:

• Client count > device count (e.g. default SCHEMA_CLIENT_NUMBER=DATA_CLIENT_NUMBER=20 with 5
  devices): every client id now gets a (possibly empty) device bucket, so SchemaClient no longer NPEs.
• Malformed/external TsFile (null device_id value, or a table with no TAG column): rows with no
  usable device identifier are skipped with a warning instead of NPEing in device-name parsing.
• Shorter/older info.txt (after REAL_DATASET_FORMAT was appended to toInfoText()):
  checkDataSet() treats a missing line as a difference instead of throwing IndexOutOfBounds.

Test Plan

• mvn -B test -pl core — full core suite green (incl. new unit + end-to-end round-trip + edge-case/NPE regression tests)
• mvn spotless:check clean
• mvn clean install -pl iotdb-2.0 -am -DskipTests builds (cross-module, shares tsfile dep)
• Round-trip test: generate TsFile dataset via writers → read back via RealMetaDataSchema + TsFileDataReader, values match
• CSV RealMetaDataSchemaTest still passes (CSV path unchanged)
• Manual smoke against a running IoTDB: generateDataMode + REAL_DATASET_FORMAT=TSFILE, then verificationWriteMode

Follow-ups (non-blocking)

• Add DataReader.close() to release TsFile handles on early stop (currently matches existing CSV reader behavior — handles freed on normal exhaustion)
• Extra test coverage: tags present, INT64/TIMESTAMP/DATE/null fields, multi-batch split, query-mode verification